The pulse profiles of the accreting X-ray millisecond pulsar SAX J1808.4-3658 at different energies are studied. The two main emission component, a black body and a power-law tail, clearly identified in the time-averaged spectrum, do not vary in phase. We show that the observed variability can be easily explained if the emission patterns of the black body and the Comptonized radiation are different: a "knife" and a "fan"-like, respectively. We suggest that Comptonization in a hot slab (radiative shock) of Thomson optical depth \~0.3 at the surface of the neutron star may be responsible for the emission. We construct a detailed model of the X-ray production accounting for the Doppler boosting, relativistic aberration and gravitational light bending. The model reproduces well the pulse profiles at different energies simultaneously, corresponding phase lags, as well as the time-averaged spectrum. By fitting the observed pulse profiles we obtain constraints on the neutron star radius (R=7.5+-1.0 km), the inclination of the system i>60 deg, and the angle between the magnetic dipole and the rotational axis delta=20deg+-5deg.